Quantization -based data hiding employing calibration and locally adaptive quantization

ABSTRACT

Certain forms of distortion make it difficult to recover hidden data embedded in an audio or image signal by quanitzation techniques. To compensate for this distortion, an embedded data reader analyzes a statistical distribution (e.g., a histogram) of feature samples ( 124 ) in an audio or image signal suspected of having hidden auxiliary data to derive an estimate of quantizers used to encode a reference signal ( 126 ). The estimated quantizers then recover the reference signal ( 126 ), and the reader uses the reference signal( 126 ) to determine and compensate for geometric or temporal distortion, like spatial scaling and rotation of image data, and time scale and speed changes of audio data. After compensating for such distortion, the reader can then more accurately recover hidden message data using quantization techniques to extract the message. The reference signal ( 126 ) is preferably repeated in blocks of the image or audio data to enable synchronization at many points in an image or audio data stream. An adaptive embedding and reading technique is used to adaptively define the quantization bins for regions of a host media signal based on signal statistics for each of the regions.

TECHNICAL FIELD

[0001] The invention relates to multimedia signal processing,steganography and data hiding.

BACKGROUND AND SUMMARY

[0002] Digital watermarking is a process for modifying physical orelectronic media to embed machine-readable auxiliary data into themedia. The media may be modified such that the embedded code isimperceptible or nearly imperceptible to the user, yet may be recoveredthrough an automated reader process. Most commonly, digital watermarkingis applied to media signals such as images, audio signals, and videosignals. However, it may also be applied to other types of mediaobjects, including documents (e.g., through line, word or charactershifting), software, multi-dimensional graphics models, and surfacetextures of objects.

[0003] Digital watermarking systems typically have two primarycomponents: an encoder that embeds the watermark in a host media signal,and a decoder that detects and reads the embedded watermark from asignal suspected of containing a watermark (a suspect signal). Theencoder embeds a watermark by altering the host media signal. Thereading component analyzes a suspect signal to detect whether awatermark is present. In applications where the watermark encodesinformation, the reader extracts this information from the detectedwatermark.

[0004] Several particular watermarking techniques have been developed.The reader is presumed to be familiar with the literature in this field.Particular techniques for embedding and detecting imperceptiblewatermarks in media signals are detailed in the assignee's co-pendingapplication Ser. No. 09/503,881 and U.S. Pat. No. 6,122,403 which arehereby incorporated by reference.

[0005] One category of data hiding methods embeds auxiliary data byquantizing samples or other features of a media signal such that thequantized sample falls within a quantization bin corresponding to adesired auxiliary message symbol. The message symbols may be binary orM-ary. Preferably, the modification made to the sample to move it intothe quantization bin is constrained to stay within a desired perceptualdistortion tolerance. This perceptual distortion tolerance enables theauxiliary data to be hidden in a manner that is imperceptible to theviewer or listener of embedded media content. One form of such auxiliarydata embedding and reading is referred to as Quantization IndexModulation or QIM. Examples of this form of data hiding is disclosed inWO9960514A1: SYSTEM, METHOD, AND PRODUCT FOR INFORMATION EMBEDDING USINGAN ENSEMBLE OF NON-INTERSECTING EMBEDDING GENERATORS and correspondingU.S. Pat. No. 6,233,347; which are hereby incorporated by reference.

[0006] To read embedded data, the samples of content expected to includeauxiliary data are evaluated to determine which quantizer orquantization bin they represent. Symbol errors may result when samplesget mapped to a different quantizer or quantization bin relative to theone used to encode them. Error correction encoding tends to mitigatesymbol errors if the distortion is not significant or does not impactthe embedded data adversely.

[0007] Quantization embedding techniques may be performed on samples orother signal features in various signal or transform domains ofmultimedia signals. To reduce the impact of distortion, the embeddingdomain may be chosen such that samples in that domain are relativelyrobust or invariant to expected types of distortion.

[0008] A challenge associated with data hiding methods, includingquantization embedding techniques, is synchronizing the reader with theembedded data after the host signal carrying the embedded data has beendistorted, e.g., geometrically or temporally distorted. Such forms ofdistortion have the effect of distorting the quantization bins. Thus, ifthe same quantizers or bins are used to interpret embedded data symbolsin the reader as in the embedder, the reader may map distorted samplesto the wrong quantizer or bin. Additional types of distortion thatspecifically effect quantization embedding techniques include theaddition of an unknown offset or multiplication by an unknown gain.

[0009] The invention provides methods of synchronizing embedded datareaders in quantization embedding systems for multimedia content. In theembedder, a reference signal and message symbols are encoded using aquantization embedding method. In the reader, the distorted data samplesare evaluated to derive quantizers or quantization bins. The reader thenuses the derived quantization information to convert the distorted datato raw symbol estimates. These raw symbol estimates are then evaluatedto detect the presence of the reference signal. Once detected, thereference signal is used to determine the distortion and compensate forit by adjusting the quantization information or distorted data such thatmessage symbols may be decoded more accurately.

[0010] Certain forms of offset and gain distortion make it difficult torecover hidden data embedded in an audio or image signal by quantizationtechniques. To compensate for this offset and gain distortion, anembedded data reader analyzes a statistical distribution (e.g., ahistogram) of feature samples in an audio or image signal suspected ofhaving hidden auxiliary data to derive an estimate of quantizers used toencode a reference signal (deriving an estimate of the quantizers isequivalent to estimating the offset and gain distortions). The estimatedquantizers then recover the reference signal, and the reader uses thereference signal to determine and compensate for geometric or temporaldistortion, like spatial scaling and rotation of image data, and timescale and speed changes of audio data. After compensating for suchdistortion, the reader can then more accurately recover hidden messagedata using quantization techniques to extract the message. The referencesignal is preferably repeated in blocks of the image or audio data toenable synchronization at many points in an image or audio data stream.

[0011] The invention provides a method of hiding auxiliary data in amedia signal such that the auxiliary data is humanly imperceptible yetrecoverable by an automated auxiliary data reader. This method comprisessegmenting the media signal into regions, determining statistics for theregions, and adapting quantization bins for each region based on thestatistics calculated for the region. To hide auxiliary data in theregions, the method quantizes signal characteristics in the regions intothe quantization bins adapted for the regions. The quantization binscorrespond to auxiliary data symbols and the signal characteristics arequantized into selected bins depending on the auxiliary data symbol tobe embedded in the signal characteristics.

[0012] Another aspect of the invention is a compatible reading methodfor this hidden data embedding method. The reading method segments themedia signal into regions, determines statistics for the regions, andbased on the statistics for each region, adapts quantization bins forthe region. To extract auxiliary data hidden in the regions, the methodmaps signal characteristics in the regions into the quantization binsadapted for the regions. The quantization bins correspond to auxiliarydata symbols, and thus symbols are extracted depending on which bins thesignal characteristics map into. In the embedding method, the signalcharacteristics have been quantized into selected bins depending on theauxiliary data symbols embedded in the signal characteristics.

[0013] Further features will become apparent with reference to thefollowing detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014]FIG. 1 is a diagram of a quantization type steganographic embedderthat embeds a hidden reference signal in media signals forsynchronization.

[0015]FIG. 2 is a diagram illustrating an example of quantizers forembedding binary symbols of a hidden reference signal by quantizing hostsignal features into quantization bins corresponding to the binarysymbols.

[0016]FIG. 3 is a diagram of a quantization type steganographic readerthat uses the hidden reference signal to synchronize reading operations.

[0017]FIG. 4 is a diagram of a histogram used to estimate quantizersfrom a distorted or corrupted media signal suspected of being embeddedwith hidden data by a quantization type steganographic embedder.

[0018]FIG. 5 illustrates an affine transformation of a circle in a planeinto an arbitrary ellipse.

[0019]FIG. 6 is a flow diagram illustrating an example of a watermarkembedder that embeds a substantially imperceptible circular referencesignal into a host image.

[0020]FIG. 7 is a flow diagram illustrating an example of a circularreference signal in the Fourier magnitude domain.

[0021]FIG. 8 is a flow diagram illustrating a detector compatible withthe embedder of FIG. 6.

[0022]FIG. 9 is a flow diagram illustrating a quantization basedembedding scheme where the quantization bins are locally adapted to hostsignal characteristics.

[0023]FIG. 10 shows an example of an image segment comprised of a threeby three pixel region.

[0024]FIG. 11 illustrates a flow diagram of a hidden data readercompatible with the embedder of FIG. 9.

DETAILED DESCRIPTION

[0025] Synchronization in Quantization Based Data Hiding Systems

[0026] This section describes techniques for synchronizing auxiliarydata readers in quantization based data embedding systems.

[0027]FIG. 1 is a diagram illustrating a quantization type data embedderthat embeds a reference signal for synchronization. The embedderreceives a host media signal such as a block of audio or image data. Forthe sake of illustration, we describe embedders that operate on samplesin the spatial domain of a digital image block (e.g., luminance samples)and the time domain of a digital audio block. Alternatively, the blockof image data may be transformed into samples in a DCT, DWT, DFT,Fourier Mellin, Gabor, autocorrelation or some other transform domain.The block of audio data similarly may be transformed into subbands,frequency coefficients, a time-frequency spectrogram, autocorrelationdomain, etc. In such cases, the quantization embedder modifies samplesin the transform domain, and an inverse transform returns the data toits perceptual domain for rendering in a form that is visible or audibleto humans.

[0028] In some implementations, the embedder may be integrated withaudio or image compression operators that transform the signal into afrequency domain, remove data in a lossy quantization process, andperform some form of entropy coding on the resulting data block. In thiscase, the quantization type data embedder operates on the frequencydomain data after lossy compression.

[0029] In an embodiment for images, the reference signal comprises atwo-dimensional array of binary symbols (1, 0). Similarly, the messagesignal comprises a two-dimensional signal of binary symbols. Thereference signal has frequency domain characteristics that facilitateits detection while minimizing its impact on perceptible distortion inthe embedded image. In particular, the reference signal comprises anarray of impulse functions in a transform domain (e.g., autocorrelationdomain, Fourier domain). These impulse functions have pseudorandomphase. When transformed to the transform domain, the reference signalhas peaks that are detected using peak detection methods (such as thosedescribed later in this document), and then aligned with reference peaksto calculate orientation parameters, such as rotation and scale.

[0030] The detected peaks are aligned with reference peaks usingcorrelation filtering.

[0031] In an embodiment for audio data, the reference signal compriseseither a one-dimensional or a two-dimensional array of binary symbols(1, 0). The two dimensional form refers to an implementation where theaudio block is transformed to a 2D spectrogram, and the samples in the2D reference signal correspond to samples in the 2D spectrogram.Similarly, the message signal comprises either a one or atwo-dimensional signal of binary symbols. The reference signal hascharacteristics that facilitate its detection while minimizing itsimpact on perceptible distortion in the embedded audio block. In oneimplementation, the reference signal comprises a PN sequence. Like theimage case above, the reference signal may have transform domaincharacteristics that facilitate detection of the reference signal in atransform domain.

[0032] For an example of embedding data in a spectrogram of an audiosignal, see U.S. patent application Ser. No. 09/661,900, filed Sep. 14,2000, entitled WATERMARKING IN THE TIME-FREQUENCY DOMAIN, which ishereby incorporated by reference. Quantization based embedding andreading of steganographic data as described in this document and thoseincorporated by reference may be performed in the time-frequency domainas detailed in application Ser. No. 09/661,900. In particular, thecalibration and/or synchronization signals described in Ser. No.09/661,900 may be embedded and read using the techniques described inthis document.

[0033] The message signal is preferably error correction encoded. Someexamples of error correction codes include convolutional codes, turbocodes, and BCH codes.

[0034] The message signal may be integrated with the reference signal,such as by convolving or multiplying it with the reference signal. Forexample, the reference signal may serve as a carrier signal for themessage signal, where the reference signal is used to spread spectrummodulate the message signal in a multiplication, convolution, exclusiveOR or exclusive NOR operation combining the message and referencesignal. For example, the message signal may be formed into a signal thathas reference peaks in a transform domain, such as the autocorrelationdomain or Fourier domain. These reference peaks serve as a referencesignal to enable detection by correlating a transformed version ofextracted message data with a known reference signal.

[0035] The message and reference signal may be embedded in mutuallyexclusive samples of the host media signal such that the two signals donot overlap. Alternatively, the message and reference signal may beembedded in feature sample domains that overlap.

[0036] The quantization type embedder reads the reference and messagesignals and maps elements in these signals to corresponding samples inthe host signal block. If the message and reference signals are combined(e.g., by multiplication or convolution), then the embedder operates onthe elements of the composite message and reference signal. It thenquantizes the corresponding host sample according to the binary symbolto be encoded. One binary symbol corresponds to a first quantizer, whilethe other binary symbol corresponds to a different quantizer. Theselected quantizer modifies the corresponding sample to fall within abin for that quantizer that minimizes the distortion to the sample.

[0037] The above auxiliary data embedders trade-off perceptualdistortion and robustness by adjusting the size and spacing betweenquantization bins. These embedders may be implemented using uniformscalar quantizers. For example, each binary symbol is associated with aquantizer having bins of width delta and offsets relative to bins of theother symbol of delta/2. This simple form of quantizer facilitates thederivation of the quantizer parameters in the reader.

[0038] Different types of quantizers may be used for the reference andmessage signals. In particular, the reference signal is embedded using ascalar quantizer, while the message signal is embedded with a vectorquantizer (e.g., a trellis coded vector quantizer).

[0039]FIG. 2 shows an example of quantizers for embedding a binaryreference signal by quantizing features of the host media signal. Onequantizer has a series of bins of size delta that correspond to a firstbinary symbol, while the other quantizer has a series of bins of sizedelta, offset by a distance delta/2 relative to the first quantizer,that correspond to a second binary symbol. The host signal feature,which has a value varying over some dimensions, such as time, space,time and space, frequency or frequency band, etc. is quantized atdiscrete samples to fall within a quantization bin corresponding to thereference signal values to be encoded in those samples. In this example,the reference signal comprises the sequence 1101001, and is repeated inthe host signal features of the block being encoded.

[0040] The reference signal does not have to be represented as a binarysequence. For example, if the implementer wants the reference signal tovary smoothly over time or space, he can choose a larger symbolalphabet, where each symbol has a corresponding quantizer. Each of thesequantizers can be represented as another quantizer axes in FIG. 2.

[0041]FIG. 3 is a diagram illustrating an auxiliary data reader thatuses the reference signal to compensate for geometric/temporaldistortions in an embedded media signal. The embedded media signal isreceived from a distortion channel, sampled and segmented into blocks.The distortion channel imposes geometric and/or temporal distortions onthe embedded media signal.

[0042] First, the reader derives the quantizers for the embedded mediasignal. In an embodiment for images, the reader performs a statisticalanalysis of the watermarked feature samples to produce a statisticaldistribution of the samples from which quantization bins can beestimated. In particular, it creates a histogram for a block of sampledimage data, such as the one shown in FIG. 4. The histogram assigns imagefeature samples to bins based on their values. The spacing between thebins provides an approximate spacing for the bins of the quantizers ofthe symbols used to encode the reference signal. It is not yet clearwhich quantizer corresponds to which binary symbols.

[0043] As noted above, the relative distance between bins in theembedder is delta/2. Due to the distortion of the watermarked signal,the distance between bins is now alpha*(delta/2), where alpha is afactor that represents the scaling of the watermarked signal due todistortion.

[0044] Based on an analysis of the histogram, the reader assumes a firstassignment of quantizer to symbol, and then applies the quantizers tothe samples to get raw estimates of the embedded symbols. It thensearches for the reference signal in the output of the quantizers. Inparticular, it correlates the reference signal with the signal formed bythe raw estimates to detect the presence of the reference signal in theembedded media signal.

[0045] As a specific example for images, consider the case where thereference signal forms peaks in the Fourier magnitude domain. The readerperforms an FFT on the image formed by the raw estimates to get Fouriermagnitude data and then transforms the data to log polar coordinates.The reader correlates the transformed data in the log polar space with alog polar representation of the reference signal. The location ofmaximum correlation provides an estimate of scale and rotation.

[0046] In a general case, if the correlation peak is not greater than athreshold, such as an absolute threshold or a threshold relative to thenext highest peak, then the reader tries the alternative quantizersymbol assignment. If neither quantizer assignments yield an acceptableamount of correlation with the reference signal, then the readerconsiders embedded data not to be present. In the special case where thedifference between one quantizer assignment and another is a phaseshift, then a correlation process need not be executed for each possiblequantizer assignment because the correlation is performed in the Fouriermagnitude domain, where phase information is not used.

[0047] If the reference signal is deemed to be present, the reader usesthe rotation and spatial scale distortion parameters to realign theimage block. A process of correlation-based matched filtering can beused, along with knowledge of the embedded reference signal phases, toestimate the translation distortion undergone by the embedded image. Thequantizers used to encode the message symbols are then used to decodethe embedded message signal. As noted, this may include generating rawsymbol estimates for samples in the block and then performing errorcorrection decoding to recover the message signal.

[0048] A similar approach may be used for audio signals. Consider anexample where the reference signal is a pseudorandom number (PN)sequence of binary symbols. The reader uses the derived quantizers toconvert the audio block to raw symbol estimates. It then correlates thePN sequence with the raw symbol estimates to determine whether thereference signal is present in the media signal. The reference signalhas peaks in the frequency domain to ascertain time scale. Inparticular, the signal formed from the raw estimates is transformed intoa frequency domain. If the reference signal is present, there should betell-tale peaks, which could be shifted due to time scale modifications(time scale modification, linear speed increases/decreases). Theposition of the peaks relative to the reference peaks provides anestimate of the time scaling distortion. Then any time-shift distortionmay be estimated by correlating the reference signal to the (re-scaled)media signal. The reader uses this estimate to re-align the audio dataand then applies the quantizers used for message symbols to extract themessage signal.

[0049] The above examples refer to quantizing image sample or audiosample features to encode a reference signal and a message signal. Thereference and message signal may be encoded in different featuresamples, temporal/spatial resolutions, frequency bands, etc. Forexample, the reference signal may be encoded in time domain samples at aparticular sampling rate for audio, or spatial domain samples at aparticular sampling rate for images (still and video). The messagesignal, on the other hand, may be encoded in other transform domainsamples, such as subbands, wavelet coefficients, DCT coefficients, DFTcoefficients, a log polar or log-log coordinate system mapped from atime-frequency spectrogram, frequency domain, temporal domain, orspatial domain, etc. The reference and message signals can be encoded inany of a variety of domains created by transforming the signal from itsperceptual domain to another transform domain. This new domain mayrepresent features of host signal, such as power, autocorrelation,energy, phase, etc. that are particularly robust to expected forms ofdistortion, yet enable the embedded data to remain imperceptible in theperceptual domain of the watermarked signal. Preferably, the transformdomain for encoding the reference and message signals should be selectedso that the quantized features are relatively invariant or robust toexpected forms of distortion for the application of the watermarkingsystem.

[0050] Synchronization Using Curve Fitting

[0051] This section describes a method for elliptical curve fitting andrelated applications. One application is to compute affine transformparameters that describe geometric transformation to an image. Inparticular, this approach inserts a circular pattern (or multiplecircular patterns) into an image, and after the image undergoesgeometric transformation distorting the circle, the elliptical curvefitting technique fits the distorted circle to an ellipse and derivesthe affine transform parameters that characterize the distortion. Theseparameters may then be used to reverse the distortion and return theimage to its original geometric state. This particular method may beused in digital watermarking applications, where a watermark encoderembeds the circular pattern in a manner that makes it imperceptible orsubstantially imperceptible to a viewer of the image. A watermarkdecoder then recovers elements of the circular pattern, fits them to anellipse and derives the affine transform parameters.

[0052] A circle in a plane has the property that any affinetransformation applied to the plane will transform the circle into anellipse. A linear transformation (i.e., without translation) applied tothe circle centered at the origin will transform it into an ellipse alsocentered at the origin. Another property of a circle pattern is that itis rotationally invariant. The advantage of this property is thatestimation of rotation can be delayed until the rest of the affineparameters (scale, differential scale and shear) have been estimated.

[0053] An equation of a circle in a plane is,

x ² +y ² =r ²

[0054] where, r is the radius of the circle.

[0055] Applying an affine transformation (with zero translation) to thecircle can be represented by a change in the coordinate system$\begin{bmatrix}x \\y\end{bmatrix} = {\begin{bmatrix}c_{1} & c_{2} \\c_{3} & c_{4}\end{bmatrix}\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix}}$

[0056] The parameters c_(i) determine the affine transformation.

[0057] Substituting the expressions for x and y from Equation 2 intoEquation 1, and simplifying, one gets the equation${\frac{x^{\prime 2}}{( \frac{r^{2}}{c_{1} + c_{3}} )} + {( \frac{{c_{1}c_{2}} + {c_{3}c_{4}}}{r^{2}} )x^{\prime}y^{\prime}} + \frac{y^{\prime 2}}{( \frac{r^{2}}{c_{2} + c_{4}} )}} = 1$

[0058] which is the geometrical equation of an ellipse. This ellipse iscentered at the origin and the major axis angle of the ellipse makes anarbitrary angle with the X-axis of the new coordinate system.

[0059]FIG. 5 illustrates an affine transformation of a circle in a planeinto an arbitrary ellipse. An automated software or hardware detectorcan compute the affine transform from knowledge of the parameters of thecircle (radius) before applying the transformation and knowledge of theparameters of the ellipse (major axis, minor axis, major axis rotation)after applying the transformation.

[0060] The following sections illustrate such a method for computing inthe context of a digital image watermark embedder and reader.

[0061] Embedder

[0062]FIG. 6 is a flow diagram illustrating an example of a watermarkembedder that embeds a substantially imperceptible circular referencesignal into a host image. The embedder starts with a host image 100. Theembedder defines a reference signal to be embedded into the image sothat the reference signal is substantially imperceptible orimperceptible (102). This reference signal is typically pre-defined, butit may also be adapted to the host image or to an auxiliary message(such as an N bit binary or M-ary code) to be embedded into the image.

[0063] The reference signal is comprised of elements that form a circlein a particular domain, such as the spatial image domain, the spatialfrequency domain, or some other transform domain. For the sake of anexample, consider a case where the elements are impulse or deltafunctions in the Fourier magnitude domain. The reference signalcomprises impulse functions located at points on a circle centered atthe origin of the Fourier transform magnitude. The points are randomlyscattered along the circle, while preserving conjugate symmetry of theFourier transform. The magnitudes of the points are determined byvisibility and detection considerations. To obscure these points in thespatial domain and facilitate detection, they have known pseudorandomphase with respect to each other. The pseudorandom phase is designed tominimize visibility in the spatial domain.

[0064] In this implementation, the definition of the reference signalonly specifies that the points should lie on a circle in the Fouriermagnitude domain. For this circular grid, the choice of the radius ofthe circle and the distribution of the points along the circle areapplication specific. For example, in applications dealing with highresolution images, the radius can be chosen to be large such that pointsare in higher frequencies and visibility in the spatial domain is low.For a typical application, the radius could be in the mid-frequencyrange to achieve a balance between visibility requirements andsignal-to-noise ratio considerations.

[0065] The distribution of points along the circle can be random orbased on a quadrant or octant symmetry as needed by the application.Points on the circle should be selected such that they are as close tothe integer lattice as possible. This will reduce the errors during theellipse fitting and estimation process required for determining theaffine transform parameters.

[0066] For this particular approach, it may be more advantageous to havefewer points with higher energy than to have several grid points withlower energy. Higher energy points will be easier to locate and isolatein the Fourier magnitude domain of a watermarked image. FIG. 7illustrates an example of a circular reference signal in the Fouriermagnitude domain.

[0067] The embedder embeds the reference signal into the image (104).One way to embed the reference signal is to add it to the host image inthe spatial image domain, or in the Fourier domain. To reduce theperceptibility of the reference signal, the samples of the signal in thespatial or spatial frequency domain may be perceptually adapted to thehiding characteristics of the host image. For example, the samples maybe multiplied by corresponding elements in a gain vector whose elementsare functions of the signal activity in a local neighborhood of samplesaround those being modified by the embedding of the reference signal.The greater the signal activity (e.g., busyness or texture), the betterthe data hiding characteristics of the image. Preferably for digitalwatermarking applications where perceptibility is a design constraint,the gain vector elements are a function of sensitivity to the humanperceptual system. The lower the sensitivity, the higher the gain.

[0068] Another way to embed the reference signal is to quantize samplevalues in the image according to an embedding function that defines howelements in the reference signal are to be mapped quantization levels orquantizers. For example, the presence of a reference signal element maybe set at one quantizer or set of quantization levels, and the absenceof a reference signal element may be set to a second quantizer or set ofquantization levels. This embedding may be performed using scalarquantizers on spatial domain samples, for example. To encode thereference signal, a sample or group of samples at a particular elementin a circular pattern are quantized using a first quantizer. The othersamples are quantized using a second quantizer.

[0069] Regardless of the specifics of the embedding process, the resultis a marked image (106) carrying the reference signal in the form of acircular pattern. The embedding process may be repeated in discreteblocks of the image. For example, the image may be tiled into 256 by 256pixel blocks at particular resolution, like 300 or 600 dpi, and theembedder may embed the reference signal into each tile.

[0070] Detector

[0071]FIG. 8 is a flow diagram illustrating a detector compatible withthe embedder of FIG. 6. The detector receives a potentially marked image120, for example, by scanning an image of a printed object suspected ofhaving a marked image. The scanning process, along with other imageprocesses, typically introduce some geometric distortion represented asan affine transform, or at least locally affine (e.g., piece wise lineargeometric distortion).

[0072] For the sake of illustration, consider the case where theembedder inserted the reference signal as a circular pattern of impulsefunctions in the Fourier magnitude domain. In this case, the detectortransforms the image into the Fourier magnitude domain (122). Of course,if the embedder defines the circular pattern in some other domain, thedetector transforms the image into that domain.

[0073] Next, the detector detects elements of the reference signal byidentifying the highest magnitude impulse signals in the transformdomain (124). This process may include one or more pre-processing stepsto enhance detection of the circular elements and differentiate themfrom the host image. Examples of these steps include summing blocks ofthe image and filtering the samples in the image to de-correlate thereference signal from the host image. Blocks may be summed in atransform domain, such as the Fourier domain, depending on the domain inwhich the reference pattern is embedded. For example, if the circularpattern is in the Fourier magnitude domain, the detector may sum blocksin the Fourier magnitude domain. The detector knows the characteristicsof the reference signal (126), such as the frequencies, number, andlocation of the elements in the circular pattern. It uses this criteriato pre-process the image and identify candidate points in the pattern.

[0074] First, a filter is used to de-correlate the reference signal fromthe host image. The detector applies this filter to the image and thencomputes the Fourier magnitude of the resulting image. The filter helpsin accentuating reference signal points.

[0075] One example of this type of de-correlating filter operates asfollows. For each image sample, it compares the sample with each of itseight neighboring image samples. The filter replaces the value at thecenter sample with a value that is incremented each time the centersample value is greater than a neighbor value and decremented each timethe center sample is less than the neighbor value. In particular, foreach comparison, the filter increments by a value of 1 if the centersample is greater than its neighbor, it increments by a value of −1 ifthe center sample is less than its neighbor, and makes no changeotherwise. The output of the filter will be between −8 and +8 when aneight neighborhood (3 by 3 sample region) is used in the filterimplementation. Such type of a filter has a number of applications suchas edge detection, signal enhancement, etc. in signal processing andoperates on different media types (image, video and audio) and samplesin various domains. For digital watermark applications, it may be usedto estimate the original host signal and watermark signal, where thewatermark signal is applied as an additive, antipodal PN signal.

[0076] The filter discussed in the previous paragraph may be implementedin variety of ways. One particular implementation makes comparisonsbetween the center sample and each neighboring sample, and transformsthe result of this comparison to an increment or decrement value (e.g.,+k or −k, where k is a constant like 1, 2, 3, etc.).

[0077] The filter sums each of the increment/decrement values from eachneighbor, and then replaces the center sample value with the result ofthe summation. This type of filter can be implemented efficiently usinga look up table. For example, the comparison operation is performed bysubtracting the center sample value from a neighbor sample value toproduce a difference value (−255 to +255 for an 8 bit sample). Theresult is then fed to a look-up table, which maps the difference valueto an increment/decrement value and outputs that value. The filter sumsthe look-up table output for each neighbor in the neighborhood, andreplaces the center sample with the result of the summation. Thisneighborhood may be the eight neighbors in 3 by 3 block of samples, theadjacent samples in a one-dimensional signal, the horizontally and/orvertically adjacent neighbors in a two or more dimensional signal, etc.The size of the neighborhood may be increased as well. The look-up tablemay be used to implement a variety of non-linear filters efficiently.

[0078] In this implementation, reference signal detection comprisesextracting peaks from the Fourier magnitude data. There are numerousapproaches for peak detection and extraction. For example, one couldisolate points with high energy relative to the average energy in theFourier magnitude plane. Alternatively, one could isolate points withhigh energy in a small local region. Statistical measures such as themean, median, variance, etc. are usually employed to reliably detectpeaks. One example of a peak detection and extraction strategy isdescribed by Nicolas Devillard. See N. Devillard, “Infrared jitterimaging data reduction algorithms”, Jun. 21, 1999, which is herebyincorporated by reference. A copy of this paper is attached.

[0079] After the de-correlation process, the detector proceeds with peakextraction as follows. Let μ be the mean of all values in the Fouriermagnitude plane. Let σ be the average absolute deviation from themedian. A binary image the same size as the Fourier magnitude plane iscreated by assigning one to all pixels which have a value above T=μ+k σand zero to all other pixels. k is an adjustable parameter that dependsupon the amount of noise. In our implementation a value of k=10 wasused. The threshold T is repeatedly reduced by a factor of k σ/100 untilthe number of peaks found is just greater than N, where N is the numberof points in the circular reference signal pattern. The non-zero pointsin the binary image are the extracted peaks.

[0080] The detector then fits the extracted peaks to an ellipse (128).The particular method is a variation of the least squares approachdescribed by Fitzgibbon et al. See, A. Fitzgibbon, M. Pilu, R. Fisher“Direct least-square fitting of Ellipses”, International Conference onPattern Recognition, Vienna, August 1996; Extended version available asDAI Research Paper #794; and M. Pilu, A. Fitzgibbon, R. Fisher“Ellipse-specific Direct least-square Fitting”, IEEE InternationalConference on Image Processing, Lausanne, September 1996, which arehereby incorporated by reference. For this approach, it is beneficial toreflect the extracted points over the entire Fourier magnitude plane,taking care of the conjugate symmetry. Reflecting the points beforefeeding them to the ellipse fitting algorithm will ensure a better leastsquares ellipse fit. Note that reflection is not a necessary condition,the approach could also be applied to points located in a portion (e.g.half plane or quarter plane) of the Fourier domain. Copies of thesepapers are attached.

[0081] The ellipse fitting procedure in this implementation is based onthe least squares ellipse-fitting technique proposed by Fitzgibbon etal. This technique uses a direct least squares method to fit an ellipseto a set of data points in a plane. The technique is based on using thesingular value decomposition to minimize the algebraic distance betweenan ellipse and the data points subject to elliptical constraints. Thesolution is non-recursive and always yields an ellipse.

[0082] Once the detector has found the ellipse, it derives theparameters of the ellipse (130). From these parameters, the detectorderives affine transform parameters that represent the distortionbetween the reference circle and the distorted reference circle (i.e.the ellipse) found in the potentially marked image.

[0083] The equation of the ellipse is specified in terms of the affineparameters and the radius of the circle (which is known, since thereference signal was based on it). Till this stage, the detector ignoresthe rotation of the points. The detector knows that the points lie onthe fitted ellipse, but does not know where they lie with respect to theoriginal reference signal. A partial affine transformation can becomputed from the parameters of the fitted ellipse. We call thistransformation partial since it does not include the effects ofrotation. Next, the detector performs an inverse geometric transform ofthe candidate points into the shape of a circle (132). The detectorapplies an inverse affine transform to the image such that the fittedellipse in the Fourier magnitude domain is converted into a circle withthe same radius as the radius of the reference signal.

[0084] The detector then recovers the rotation angle parameter. In oneapproach for getting rotation, the detector samples along the detectedcircle, and performs a one dimensional correlation (134) to find therotation angle. In particular, the detector samples the Fouriermagnitude along this circle to obtain a sampled 1-dimensional signal.This 1-D signal contains peaks corresponding to the reference signalpoints. The detector gets the rotation angle (136) by a 1-D correlationbetween this signal and the 1-D signal corresponding to the referencesignal Fourier magnitude sampled along the reference signal circle. Thedetector computes the rotation angle from the correlation peak offset.

[0085] Alternatively, rotation can be recovered using a Fourier-Mellinprocedure in the log-polar domain. For this approach, the partial affinetransform estimated from the ellipse is inverted and applied to thefrequency domain data so that the extracted points lie on a circle butare rotated with respect to the reference signal. Rotation is recoveredthrough correlation in the log-polar domain. The estimated rotation,along with the partial affine parameters estimated from the ellipse arethen combined to give the full affine transform parameters.

[0086] Having determined these affine parameters, the detector may usethese parameters to correct for differential scale and rotation in thespatial domain or some other domain, and then perform additional stagesto get other parameters, such as translation. In particular, thedetector may perform phase correlation (138) (e.g., phase only matchfiltering) between the reference signal and the marked signal, correctedfor scale, shear and rotation distortion. The coordinates of highestcorrelation provide the translation parameters (140).

[0087] In the case where the watermark embedder inserts the referencesignal using quantization modulation as described above, the detectorfirst performs a quantization to convert image samples into a binaryimage representing the presence or absence of the reference signal. Thedetector then fits the detected reference signal to an elipse, andderives the affine parameters as described above.

[0088] Images having the reference signal can undergo severe degradationdue to image manipulations or noise. Such degradation can adverselyaffect the estimation of the affine parameters. Noise or degradationscan cause spurious peaks to appear in the Fourier magnitude domain. Theellipse-fitting technique of Fitzgibbon et al. is substantially affectedby spurious points that are not on the ellipse. Even a few spuriouspoints can alter the ellipse-fitting result to such an extent that theresulting affine parameters are not accurate enough for successfulwatermark detection and decoding.

[0089] There are a number of potential enhancements to the methoddescribed above. To improve the method, one implementation discardsoutliers in the ellipse fitting process as follows:

[0090] 1. start with N_(i) reference point candidates detected in stage124 of FIG. 4;

[0091] 2. get the best ellipse fit for the N_(i) points; this is ellipseE_(i);

[0092] 3. compute distance from each point to E_(i);

[0093] 4. sort N_(i) points by distance from E_(i) in descending order;

[0094] 5. discard point that is at the maximum distance from the ellipseE_(i);

[0095] 6. if the difference between the ellipse fit at the previousiteration and the current iteration is less than ∈, go to Step 9

[0096] 7. N_(i+1)=N_(i)−1

[0097] 8. increment i; i=i+1

[0098] 9. go to Step 2 for next iteration

[0099] 10. stop

[0100] Another modification to this approach includes using knowledge ofthe reference signal to discard points that are highly unlikely to bepart of the reference signal.

[0101] An initial implementation of the above method uses 12 referencepoints in a quarter plane of the Fourier magnitude space. The embeddersubdivides the image into 128 by 128 blocks and transforms the blocksinto a 128 by 128 Fourier magnitude space. As a further enhancement,each of the N points extracted in stage 124 may be assigned a weightaccording to their magnitude value and location in the Fourier transformdomain. These weights could then be incorporated within the ellipsefittting procedure to improve both the robustness as well as theaccuracy of the fit.

[0102] As a further enhancement, the reference signal may be designed tocarry auxiliary, hidden information. For example, each reference pointmay correspond to a message value. A particular sequence of referencepoints at a particular distance from each other on the reference circlemay be used to signal a start of a message. Then, the remainingreference points can be used to carry multiple bits of information. Forexample, the presence of reference points at particular locations maycorrespond to binary values. Error correction coding and decoding may beused to encode the message symbols and decode the message symbols.Multiple circles may be encoded into the content by tiling the circlesin blocks of image data (e.g., putting a circle in each block).

[0103] Further, information may be encoded by representing differentmessages using different arrangements of circles.

[0104] As a further enhancement, each tile of the watermark signal couldcontain a unique arrangement of points along a circle, with the radiusof the circle being constant in all tiles.

[0105] As a further enhancement, the reference signal may be used as anindicator of another watermark signal, or even convey key information,specifying how or where to decode another watermark signal. For example,the reference signal may be used as a calibration signal that enablesthe detector to determine the affine transform parameters, correct forthe distortion, and then decode a message signal from the image data(e.g., a spatial or frequency domain watermark message signal).

[0106] The reference signal may be used in video signals in a similarfashion. For example, the reference signal may be encoded into thespatial domain or spatial frequency domain of selected frames. Also, thereference circle may be encoded in the time-frequency spectrogram of anaudio signal.

[0107] Locally Adaptive Embedding and Reading Using Quantization

[0108] In this section, we describe a digital watermarking method thatlocally adapts quantization bins based on local characteristics of thehost signal. These quantization bins correspond to auxiliary datasymbols. To hide an auxiliary data symbol in a host media signal, theembedder alters the value of a characteristic of the host signal suchthat its value falls within a bin corresponding to the auxiliary datasymbol to be encoded. To extract the hidden data symbol, a readerre-computes the quantization bins based on local characteristics of thehost signal, maps the characteristic values at particular embeddinglocations to the locally adapted bins to determine the bin, and thenreturns the symbol corresponding to that bin. While our examplesspecifically describe a process for image or video signals, thetechnique applies to other media types like audio signals.

[0109]FIG. 9 illustrates a flow diagram of a quantization basedembedding scheme where the quantization bins are locally adapted to hostsignal characteristics. As we will explain further, our implementationuses a separate watermark to provide additional functionality. As such,the process starts with an initial watermark embedding operation toembed a first watermark signal as shown in block 148. This procedureproduces our host media signal 150, which serves as input to thequantization-based embedder.

[0110] This embedder begins by segmenting the media signal into blocks(152). These blocks may have a regular or arbitrary shape. In the formercase, the media signal may be divided into contiguous blocks of apredetermined dimension. For example, one of our implementationssegments spatial domain pixels of an image signal into rectangularblocks.

[0111] In the latter case, a segmentation routine may be used toidentify regions in the media signal having particular characteristics.For example, the routine may segment noisy from smooth regions or tonalfrom non-tonal segments. It may segment regions based on similarity offeature values using watershed or morphological segmentation routines,such as those described in co-pending patent application Ser. No.09/940,872, which is hereby incorporated by reference. One example of asuitable segmentation process is watershed segmentation. See, forexample Luc Vincent and Pierre Soille, “Watersheds in Digital Spaces: Anefficient Algorithm Based on Immersion Simulations,” IEEE Trans. PAMI,Vol13 No 6, June 1991; and Kari Saarinen, “Watersheds in Color ImageSegmentation,” IEEE Workshop on Nonlinear Signal and Image Processing(NSIP), Halkidiki, Greece, June 1995, which are hereby incorporated byreference.

[0112] Other segmentation approaches may be used as well. Segmentationmethods fall into 3 categories, (1) characteristic feature thresholdingor clustering, (2) edge detection, and (3) region extraction. See K. S.Fu and J. K. Mui, “A Survey on Image Segmentation,” Pattern Recognition,Vol 13, pp3-16, 1981, which is hereby incorporated by reference. Also,within these methods, many techniques can be employed to compute thequantities used in the methods, e.g., statistics, edge filters,morphological filters, Markov random fields (MRF), fuzzy set theory, andneural nets. See Robert Haralick and Linda Shapiro, “Survey: ImageSegmentation Techniques,” Computer vision, graphics, and imageprocessing, Vol 29, pp 100-132, 1985, which is hereby incorporated byreference.

[0113] For images, we recommend use of the watershed algorithm, whichfalls into the clustering segmentation category due to the inherentrobustness we have seen with respect to typical noise and distortionprocesses for imagery. However, other segmentation methods could beemployed for computational or robustness reasons with similar results.Furthermore, these segmentation techniques can be combined or layeredfor additional robustness in specific applications.

[0114] One potential advantage is that the host signal characteristicsselected for quantization-based embedding can be chosen based on thecharacteristics of each region. In this case, the region is mapped intoa domain comprising an array of characteristic values for that region.The quantization bins are adapted to the characteristic values of thatregion. Also, the quantization function parameters, such as the binsizes and spacing, may be selected based on those characteristics. Thequantization bin spacing, for example, may be represented as anon-linear function of the characteristic values of the host signal(e.g., logarithmic).

[0115] One benefit of selecting the signal characteristic for embeddingbased on the characteristics of the region is that the data hiding canbe optimized for increased robustness and/or reduced perceptibilitybased on those characteristics. For example, a first type ofquantization function may be used to establish the bins for a noisyregion, while a second type of quantization function may be used toestablish the bins for a smooth region.

[0116] Another potential advantage is that the host signalcharacteristics and the quantization function that defines the binstructure can be selected based on the distortion channel through whichthe watermarked signal is expected to travel. For a robust watermarkapplication, the quantization-based watermark is designed to survive thedistortion channel for that channel. As such, the signal characteristicsand quantization function are selected to survive that channel. Examplesof the distortion channel include printing and scanning, compression,digital to analog conversion (and analog to digital conversion), formatchanges, re-sampling, geometric distortion (including temporaldistortions like time scaling), etc.

[0117] After segmenting the signal, the embedder determines localstatistics of the characteristic values in each segment to be used inhiding auxiliary data (154). Continuing with our example of an imagesegmented into rectangular blocks, the embedder determines orderstatistics of selected pixels in each block, such as the minimum,maximum, and median value. From these order statistics, the embedderdefines the bin parameters, namely the bin size and spacing. Theseparameters may be specified in part by a quantization function, whichdefines quantization bins as a function of the host signalcharacteristic being modified to hide auxiliary data.

[0118]FIG. 10 shows an example of an image segment comprised of a threeby three pixel region. In this example, an image is segmented intocontiguous regions like the one depicted in FIG. 10. Each pixel locationin the region is either a reference pixel or target pixel to bequantized into a bin to hide auxiliary data. The embedder uses thereference pixel values to compute the local characteristics, such as theminimum and maximum values as well as the median or average. It thensets the quantization bin size as function of the statistics: e.g.,:K(Max-Min), where K is a constant, and Max and Min are the maximum andminimum values of the reference pixels in the region. The median oraverage may be used to set the quantizer offset. This is just oneexample; there are many ways to select the quantization function used toestablish the bins, including non-linear or linear quantizationfunctions, functions selected based on the region characteristics, etc.Preferably, the reference values, statistics, and quantization functionshould be selected such that the quantization bins are substantiallyinvariant to the expected distortions of the channel.

[0119] While our illustrated scheme employs separate reference andquantization embedding locations, these locations may overlap in someimplementations. Depending on the region size, choice of host signalcharacteristics for embedding, and the quantization function, thereference locations used to compute the statistics may be the same as orat least overlap the embedding locations. The embedding and referencelocations may correspond to groups of samples of the host signal thatare mutually exclusive yet interleaved as shown, that are partiallyoverlap, or that substantially overlap.

[0120] The reference locations may be used to embed a differentwatermark, such as a detection, calibration, variable message orsynchronization watermark. We alluded to this type of digital watermarkabove with reference to block 148 in FIG. 9. These types of watermarksmay be distinct or one in the same. Again, the embedding locations ofthe separate watermark and the quantization-based watermark may bemutually exclusive, overlap partially, or substantially overlap. Weprovide specific examples of this separate watermark in the discussionof the reader below.

[0121] Returning to FIG. 9, the embedder proceeds to embed auxiliarydata symbols by adjusting the characteristic value of an embeddinglocation so that it falls within a quantization bin associated with thesymbol to be encoded (158). Each embedding location may encode oneauxiliary binary data symbol. Alternatively, groups of embeddinglocations may correspond to an M-ary symbol in a vector quantizationscheme. For example, the block in FIG. 10 has five embedding locationslabeled Q. The characteristics of these five embedding locations can beadjusted such that they fall into one of M different bin sets, each binset corresponding to a particular M-ary symbol. Further, the auxiliarydata symbols are preferably error correction encoded prior to embeddinginto the host signal. The result of the quantization based embeddingoperation is a marked signal (160).

[0122] The marked signal passes through a channel for a particularapplication, and ultimately arrives at a compatible reader. This channelis expected to have some types of distortion that increase thedifficulty of accurately recovering the embedded auxiliary data.

[0123]FIG. 11 illustrates a flow diagram of a reader compatible with theembedder of FIG. 9. The reader takes as input a watermarked signal 160.It then executes procedures to detect, calibrate and extract the hiddenauxiliary data. First, the reader detects whether a digital watermark ispresent (162). This may be accomplished by detecting the presence of adetection watermark as noted above. Preferably this detection watermarkis robust to distortions expected in the channel. For example, for imagesignal applications, the robust watermark is preferably robust torotation and scale. One form of detection watermark is a hidden signalembedded in the Fourier Mellin domain, which is invariant to rotationand scale changes. This attribute enables the reader to identify thepresence of a digital watermark. The detection watermark may be aseparate watermark, or a watermark signal integrated with anotherwatermark.

[0124] Having detected the presence of the digital watermark, the readerproceeds to calibrate the signal (164). In this calibration process, thereader uses attributes of the watermark signal (e.g., known signalcomponents) to determine the origin and coordinate system of thewatermark signal. This process is sometimes referred to as calibrationor synchronization of the watermark, and may be performed with areference, calibration, or synchronization watermark signal. Thisprocess enables the reader to identify the reference and embeddinglocations used in the quantization based embedding process. A separatewatermark, such as the detection, reference, synchronization, or acalibration watermark, may be used to determine the origin andcoordinate system. For images and video, the calibration processdetermines the translation, scale, rotation, and differential scale ofthe host signal relative to its original state at embedding using knownattributes of the watermark signal to align the data in the receivedsignal. For audio, the calibration process identifies the translationand time scaling. For some distortion channels, certain types ofgeometric distortion are not likely, so the calibration process can besimplified to merely finding the origin and/or geometric or temporalscaling. For example, rotation may not occur in certain video and audioapplications. Thus, the watermark need not be robust to rotationdistortion.

[0125] There are a number of ways to perform the calibration function.One such approach is to use the elliptical curve fitting methoddescribed in this document. Another is to perform correlation detectionwith peaks of the watermark signal in the Fourier Mellin domain (orother domain, such as the Fourier domain, autocorrelation domain, etc.)to get rotation and scaling parameters, and then perform additionalcorrelation to get translation parameters (e.g., the origin of awatermarked block). Such techniques are described in U.S. Pat. No.6,122,403, or co-pending application Ser. No. 09/503,881, which areincorporated by reference.

[0126] Once the geometric distortion parameters are computed, the readercan align the received signal and evaluate the embedding locations inthe aligned signal of the quantization based watermark.

[0127] Next, the reader repeats the processes performed in blocks152-156 of the embedder to segment the aligned signal, compute the localstatistics of each segment (e.g., optionally on the reference locationsin the particular domain of quantization based embedding), and determinethe quantization bin parameters (166). The reader then decodes thequantization based watermark signal by mapping the signalcharacteristics at the embedding locations into the re-computedquantization bins (168). The bins each correspond to one M-ary or binarysymbol. Thus, by mapping the signal characteristics into the bins (e.g.,applying the quantization function to the signal characteristic), thereader determines the symbol for each embedding location.

[0128] These symbols may have been error correction encoded, in whichcase, they are supplied to an error correction decoder to extract thehidden auxiliary data message.

[0129] One potential drawback for some quantization-based digitalwatermark embedding schemes is that they are not robust to gain or biasdistortions. In other words, if a gain is applied to the watermarkedsignal, or if a bias signal is added to it, the signal characteristicsthat have been quantized to hide auxiliary data symbols may be alteredsuch that they no longer correspond to the correct data symbols. Byadapting the quantization function in this manner, thequantization-based digital watermark may be made more robust. Further,since the bin structure varies with the host image, it enables data tobe hidden more securely.

[0130] Quantization Based Watermarks Combined With Other Watermarks

[0131] The quantization based embedding and reading method describedabove enables a number of possible applications. It may be used to embedinformation separate from another watermark in the same host mediasignal. One example is sync or calibration watermark describedpreviously. The other digital watermark or watermarks may be embeddedand extracted using techniques like direct sequence or frequency hoppingspread spectrum.

[0132] Attributes of a first watermark embedded into the host mediasignal may be used to determine the quantization bins used to embed asecond digital watermark. For example, a calibration signal comprised ofsignal peaks in the autocorrelation or Fourier magnitude domain may beused as a component of the first watermark. This watermark isperceptually adapted to the host signal based on its signal activity,variance, or local contrast, where greater signal activity yields moreintensity in the first watermark signal. The statistics of this firstwatermark over a specified local region, such as its root means square,can be used to set the quantization bin sizes for the quantization basedwatermark embedded in a second watermark in that local region. As theRMS of the first watermark increases, so does the bin size of thequantization based watermark and its robustness.

[0133] Another type of digital watermark is one that is generated byspreading a message over pseudorandom number carrier sequence and addingthe resulting signal to the host media signal with or without adaptingit based on human perceptibility models. Such watermarks may be detectedusing correlation based detection techniques, where the received signalis correlated with the carrier sequence. The addition of such awatermark before embedding using the quantization based approach addsvariability to the host signal, enabling the quantization based hiddendata to be embedded more robustly.

[0134] Additional types of digital watermarks that may be embedded alongwith the quantization-based watermark are disclosed in assignee's U.S.Pat. No. 6,122,403 and other literature. Such digital watermark signalsmay carry a key to extract the quantization-based watermark. Forexample, the key may specify the embedding locations, the signalcharacteristics that have been quantized to encode hidden data, or thegeneral form of the quantization function (the local statistics can thenbe used to identify the quantization bins within that general form withmore particularity).

[0135] The quantization-based approach may be used for steganographic ordata hiding applications, where the objective is to hide larger amountsof information. A first digital watermark is embedded using, forexample, spread spectrum techniques and correlation based detection toincrease robustness. This robust watermark carries a small payload, orno payload, and performs the function of a detection and/or calibrationwatermark signal. The quantization watermark carries additional data andprovides a higher capacity hidden data channel for auxiliary informationwithin the host media signal (e.g., video, still image or audio).

[0136] Another application for dual watermarks, including quantizationbased watermarks and other non-quantization watermarks is “metrology.”Metrology refers to the science of measurement. In the context of thedigital watermark, it refers to measurements of the digital watermark ina distorted host media signal to analyze the channel through which thehost media signal has passed. These measurements can be used tospecifically identify types of distortions or alterations to the hostsignal due to the channel to authenticate the signal or otherwisecharacterize the channel.

[0137] One authentication application for watermarked objects, such asvalue documents and identification cards, is to detect whether theobject is a copy or an original based on the watermark measurements.

[0138] One specific framework for characterizing the channel distortionis as follows. The above quantization based watermark is used to embed ahidden message, which has been error correction encoded (e.g., usingconvolution codes or some other scheme like turbo or Reed Solomoncodes). A programmatic watermark decoder function operates on the hostsignal after it has passed through a distortion channel (or model ofthat channel) and extracts raw, error correction coded messages bymapping characteristics to quantization bins. The decoder thenreconstructs the original message by performing error correctiondecoding. It then reconstructs the original raw message symbols byperforming a message coding on the extracted message. This message neednot be fixed; it can and preferably does carry variable data payloads.Once the raw message symbols are reconstructed, the decoder computeserror statistics by comparing the reconstructed raw message symbols withthe ones identified by mapping the received signal characteristics tothe quantization bins. When the detected raw symbol differs with thereconstructed symbol, it is identified as an error.

[0139] Error statistics may be maintained per bin, per region, and/orper signal characteristic. A training set of host media signals (such asimages used on a secure document) can be used to characterize errors dueto distortion channels. Based on this training set, the implementer cancreate a classifier that classifies distortion types (like scanning andprinting or photocopying) based on the observed error statistics fromthe training set. The classifier can then be deployed in a programmaticauthenticator that measures error statistics of the quantization basedwatermark signal and classifies the type of distortion (e.g.,differentiates an original from a copy or multiple classes of copies).

[0140] A similar approach can be used to characterize a distortionchannel and measure error statistics that provide a measure of qualityof service, such as quality of service of network video or audiobroadcasts.

[0141] One way to refine the sensitivity of the quantization-basedwatermark to certain forms of distortion is to adjust the bin sizes overthe space of the signal characteristics being quantized so that they aremore likely to generate raw symbol errors in particular bins in responseto specific forms of distortion being detected. When a high number oferrors are observed in those bins, then the resulting error measurementis an indicator of a particular type of distortion known to beassociated with that error measurement.

[0142] In some applications, it is useful to embed a hash of the hostsignal in the digital watermark. The above quantization approach can beused to embed the hash of the host signal into the host signal. A robusthash refers to a hash that is not altered due to the embedding of thehash in the host media signal. In other words, the robust hash of themedia signal is the same before and after embedding of the hash.Typically, the robust hash is derived from perceptually relevantfeatures. To detect alteration of the host signal, the robust hash isextracted and compared with a newly computed hash of the received signalfrom which it is extracted. If these two hashes do not match, analteration has occurred.

[0143] Concluding Remarks

[0144] Having described and illustrated the principles of the technologywith reference to specific implementations, it will be recognized thatthe technology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicants incorporate by reference the patents andpatent applications referenced above.

[0145] The methods, processes, and systems described above may beimplemented in hardware, software or a combination of hardware andsoftware. For example, the auxiliary data encoding processes may beimplemented in a programmable computer or a special purpose digitalcircuit. Similarly, auxiliary data decoding may be implemented insoftware, firmware, hardware, or combinations of software, firmware andhardware. The methods and processes described above may be implementedin programs executed from a system's memory (a computer readable medium,such as an electronic, optical or magnetic storage device).

[0146] The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patents/applications are also contemplated.

1-50. (cancelled)
 51. A method of embedding a digital watermark in amedia signal comprising: embedding a first digital watermark in themedia signal to produce a watermarked media signal; quantizing elementsof the watermarked media signal to values corresponding to messagesymbols of a secondary digital watermark; wherein the first digitalwatermark is used to calibrate the media signal before extracting thesecondary digital watermark.
 52. The method of claim 51 whereinstatistics of the first digital watermark are used to adapt thequantizing of elements to embed the secondary watermark.
 53. The methodof claim 51 wherein the first digital watermark includes a directsequence or frequency hopping spread spectrum digital watermark.
 54. Themethod of claim 51 wherein the first watermark is generated by spreadinga message over a carrier signal to produce a modified carrier and isembedded by adding the modified carrier to the media signal to producethe watermarked media signal.
 55. The method of claim 51 wherein thefirst watermark carries a key to extract the secondary watermark. 56.The method of claim 55 wherein the key specifies watermark parametersused in quantizing the watermarked signal to embed the secondarywatermark.
 57. The method of claim 51 wherein the first digitalwatermark includes a reference signal having a circular pattern that isused to compensate for geometric distortion using elliptical curvefitting.
 58. A tangible medium on which is stored instructions forperforming the method of claim
 51. 59. A method of decoding a digitalwatermark from a watermarked media signal comprising: detecting a firstdigital watermark in the watermarked media signal, the first digitalwatermark enabling calibration of the watermarked media signal; andquantizing elements of the calibrated watermarked media signal toextract values corresponding to message symbols of a secondary digitalwatermark; wherein the first digital watermark is used to calibrate thewatermarked media signal before extracting the secondary digitalwatermark.
 60. The method of claim 59 wherein the first digitalwatermark includes a direct sequence or frequency hopping spreadspectrum digital watermark.
 61. The method of claim 59 wherein the firstwatermark is generated by spreading a message over a carrier signal toproduce a modified carrier and is detected using correlation with thecarrier signal.
 62. The method of claim 59 wherein the first watermarkcarries a key to extract the secondary watermark.
 63. The method ofclaim 62 wherein the key specifies watermark parameters used inquantizing the watermarked signal to embed the secondary watermark. 64.The method of claim 59 wherein the first digital watermark includes areference signal having a circular pattern that is used to compensatefor geometric distortion using elliptical curve fitting.
 65. A tangiblemedium on which is stored instructions for performing the method ofclaim 59.